Learn to Track: From Images to 3D Data

Object pose estimation aims at determining the relation between the camera and the object of interest. For rigid objects, the pose is defined through three degrees of freedom for rotation and three for translation. Across different applications in robotic perception, augmented reality and human-computer interaction, achieving a real-time robust object pose estimation is an integral component to reach their goals. This thesis focuses on object pose estimation using a temporal tracker from 3D data. The objective is not only to achieve robustness through public benchmarks, but also to integrate into a wide-range of real-world applications. A temporal tracker is a frame-to-frame algorithm that relays the object’s pose from the previous frame and updates it for the current frame. When we investigate the use case of temporal trackers, they are frequently applied on reconstruction frameworks to track a single object or a static scene. However, when tracking several independently moving objects present in the scene, most frameworks rely on tracking-by-detection. Conversely, tracking-by-detection methods look at each frame independently. For each frame, they localize the object on the image and estimate the pose. On one hand, the limitation of a temporal tracker is the requirement to initialize on the first frame either manually or by a detector. On the other, the benefit of a temporal tracker is its efficiency because it simplifies the problem to updating the pose between two consecutive frames. Intuitively, the combination of both approaches, where the temporal tracker is initialized by the detector and then temporally track the objects, is a natural procedure. But, to be robust against clutter, occlusions and fast movements, most temporal trackers require at least 100 ms per frame with a GPU optimization. Since independent temporal trackers are triggered for each object in the scene, it also scales linearly to the number of objects. In effect, tracking-by-detection approaches overcome the efficiency of the temporal trackers with about five objects in the scene. As a consequence, the temporal tracker is not commonly used because it cannot uphold to its promised efficiency. Keeping this in mind, we formulate a learning-based 3D temporal tracker that learns random forests solely from a 3D CAD model to predict the transformation update from the input depth images. On average, it runs at less than 2 ms per frame for each object on a single CPU core. Hence, even if the computational complexity of the tracker is linear to the number of objects, the extremely low magnitude of its tracking time allows the algorithm to track over a hundred moving objects in a scene at 30 fps with only 8 CPU cores. This is significantly faster than any tracking-by-detection approach. Due to the tracker’s efficiency, we have successfully demonstrated the combination of detection and temporal tracking in a single framework to achieve a seamless performance, i.e. with low latency. Although the first version of the tracker works on specific object instances, meaning that the shape of the object of interest that appears in the scene has to match the geometry of the learned CAD model, we further developed an extension that generalizes the tracker to perform pose estimation for any object within a class. Through the 3D head pose estimation, this allows the generalized tracker to directly estimate the pose of an arbitrary user without any a priori information about them. Nevertheless, tracking with an object instance still attains better accuracy than a generalized tracker. Due to this, we introduce a fast and reliable calibration procedure that optimizes the shape model of a specific object from a given class. In this thesis, to